Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Extending AI Actions #2537

Merged
merged 6 commits into from
Dec 19, 2024
Merged

Extending AI Actions #2537

merged 6 commits into from
Dec 19, 2024

Conversation

mnocon
Copy link
Contributor

@mnocon mnocon commented Nov 11, 2024

Question Answer
JIRA Ticket JIRA: https://issues.ibexa.co/browse/IBX-9077
Versions 4.6 and 5.0
Edition Headless, Experience, Commerce

This PR builds upon the work done in #2473 - and adds the "how to extend" part of the doc, together with references.

I've included the generated PHP API Reference (this is stolen from #2447) - to avoid polluting this PR with too many files the generated reference is extracted to a separate PR (and all the preview links link there).

General preview: preview

New pages or sections:

Testing the use cases locally:

Setup

Run in a project:

git clone https://github.com/ibexa/documentation-developer -b IBX-8689-extending
cp -R documentation-developer/code_samples/ai_actions/ .
merge the ibexa_admin_ui.yaml and webpack.config.js changes together with the existing content
composer run post-install-cmd

Use case 1

  1. Configure OpenAI
  2. Publish an image without an alt text
  3. Run php bin/console app:add-alt-text

The image should have alt text generated automatically

Goal: teach readers how to use the AI PHP API

Use case 2

  1. Run llamafile on port 8080
  2. Create an Action Configuration using the LLaVATextToText Handler
  3. Use it in online editor to improve writing

Goal: teach readers how to create a custom Handler (including Action Handler Form options)

Use case 3

  1. Download sample audio files (file1, file2)
  2. Install Whisper and make it available as an executable called whisper
  3. Add a eztext field with transcript identifier to the File Content Type
  4. Create an Action Configuration using the Whisper Handler for the Transcribe Audio Action Type
  5. Start creating a new file and use the AI to generate a transcription

Goal: teach readers how to create a custom Action Type, including:

  • Action Type
  • Form for Action Type options
  • custom Handler
  • integration with the REST API
  • integration with the back office

TODO:

Checklist

  • Text renders correctly
  • Text has been checked with vale
  • Description metadata is up to date
  • Code samples are working
  • PHP code samples have been fixed with PHP CS fixer
  • Added link to this PR in relevant JIRA ticket or code PR

@mnocon mnocon changed the base branch from master to IBX-8689 November 11, 2024 23:10
@mnocon mnocon marked this pull request as ready for review November 12, 2024 08:33
Base automatically changed from IBX-8689 to master November 18, 2024 12:41
```

``` yaml
[[= include_file('code_samples/ai_actions/config/services.yaml', 33, 37) =]]

This comment was marked as outdated.

```

``` yaml
[[= include_file('code_samples/ai_actions/config/services.yaml', 38, 45) =]]

This comment was marked as resolved.

@mnocon mnocon force-pushed the IBX-8689-extending branch 4 times, most recently from 5967dee to 2982a8e Compare December 9, 2024 09:57
@ibexa ibexa deleted a comment from github-actions bot Dec 12, 2024
docs/ai_actions/extend_ai_actions.md Outdated Show resolved Hide resolved
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we really need the full file to finally display few lines? We won't maintain the big beginning of the file from version to version anyway.

I know it goes with executing the example code on a clean install. I prefer my "append_to_" approach like in #2195 or #2222.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've slimmed down both this file and the webpack.config.js as you suggest - I agree, it's better in the long run


private function convertImageToBase64(?string $uri): string
{
$file = file_get_contents($this->projectDir . \DIRECTORY_SEPARATOR . 'public' . \DIRECTORY_SEPARATOR . $uri);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Shouldn't this use the binary/IO Handler or similar abstract layer to be compatible with DFS or other cluster storage?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for the suggestion! Ok for you to improve this in a follow-up? I agree it would be best to show a best practise here.

I think https://github.com/ibexa/core/blob/4.6/src/lib/IO/IOBinarydataHandler.php is what I might be looking for, but I will confirm that

docs/ai_actions/extend_ai_actions.md Outdated Show resolved Hide resolved
docs/ai_actions/extend_ai_actions.md Outdated Show resolved Hide resolved
docs/ai_actions/extend_ai_actions.md Outdated Show resolved Hide resolved
Copy link
Contributor

@julitafalcondusza julitafalcondusza left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please check my comments.

docs/ai_actions/extend_ai_actions.md Outdated Show resolved Hide resolved
docs/ai_actions/extend_ai_actions.md Outdated Show resolved Hide resolved
docs/ai_actions/extend_ai_actions.md Outdated Show resolved Hide resolved
Comment on lines 12 to 13
## Execute Actions
You can execute AI Actions by using the [ActionServiceInterface](../api/php_api/php_api_reference/classes/Ibexa-Contracts-ConnectorAi-ActionServiceInterface.html) service, as in the following example:

This comment was marked as resolved.

docs/ai_actions/extend_ai_actions.md Outdated Show resolved Hide resolved
docs/ai_actions/extend_ai_actions.md Outdated Show resolved Hide resolved
Comment on lines 225 to 226
#### Handle input data
Start by creating an Input Parser able to handle the `application/vnd.ibexa.api.ai.TranscribeAudio` media type.

This comment was marked as resolved.

docs/ai_actions/extend_ai_actions.md Outdated Show resolved Hide resolved
docs/ai_actions/extend_ai_actions.md Outdated Show resolved Hide resolved
docs/ai_actions/extend_ai_actions.md Outdated Show resolved Hide resolved
Co-authored-by: julitafalcondusza <117284672+julitafalcondusza@users.noreply.github.com>
Co-authored-by: Adrien Dupuis <61695653+adriendupuis@users.noreply.github.com>
Copy link

code_samples/ change report

Before (on target branch)After (in current PR)

code_samples/ai_actions/assets/js/addAudioModule.js


code_samples/ai_actions/assets/js/addAudioModule.js

docs/ai_actions/extend_ai_actions.md@357:``` js
docs/ai_actions/extend_ai_actions.md@358:[[= include_file('code_samples/ai_actions/assets/js/addAudioModule.js') =]]
docs/ai_actions/extend_ai_actions.md@359:```

001⫶import { addModule } from '../../vendor/ibexa/connector-ai/src/bundle/Resources/public/js/core/create.ai.module';
002⫶import TranscribeAudio from './transcribe.audio';
003⫶
004⫶addModule(TranscribeAudio);


code_samples/ai_actions/assets/js/transcribe.audio.js


code_samples/ai_actions/assets/js/transcribe.audio.js

docs/ai_actions/extend_ai_actions.md@349:``` js
docs/ai_actions/extend_ai_actions.md@350:[[= include_file('code_samples/ai_actions/assets/js/transcribe.audio.js') =]]
docs/ai_actions/extend_ai_actions.md@351:```

001⫶import BaseAIComponent from '../../vendor/ibexa/connector-ai/src/bundle/Resources/public/js/core/base.ai.component';
002⫶
003⫶export default class TranscribeAudio extends BaseAIComponent {
004⫶ constructor(mainElement, config) {
005⫶ super(mainElement, config);
006⫶
007⫶ this.requestHeaders = {
008⫶ Accept: 'application/vnd.ibexa.api.ai.AudioText+json',
009⫶ 'Content-Type': 'application/vnd.ibexa.api.ai.TranscribeAudio+json',
010⫶ };
011⫶ }
012⫶
013⫶ getAudioInBase64() {
014⫶ const request = new XMLHttpRequest();
015⫶ request.open('GET', this.inputElement.href, false);
016⫶ request.overrideMimeType('text/plain; charset=x-user-defined');
017⫶ request.send();
018⫶
019⫶ if (request.status === 200) {
020⫶ return this.convertToBase64(request.responseText);
021⫶ }
022⫶ else {
023⫶ this.processError('Error occured when decoding the file.');
024⫶ }
025⫶ }
026⫶
027⫶ getRequestBody() {
028⫶ const body = {
029⫶ TranscribeAudio: {
030⫶ Audio: {
031⫶ base64: this.getAudioInBase64(),
032⫶ },
033⫶ RuntimeContext: {},
034⫶ },
035⫶ };
036⫶
037⫶ if (this.languageCode) {
038⫶ body.TranscribeAudio.RuntimeContext.languageCode = this.languageCode;
039⫶ }
040⫶
041⫶ return JSON.stringify(body);
042⫶ }
043⫶
044⫶ afterFetchData(response) {
045⫶ super.afterFetchData();
046⫶
047⫶ if (response) {
048⫶ this.outputElement.value = response.AudioText.Text.text[0];
049⫶ }
050⫶ }
051⫶
052⫶ toggle(forceEnabled) {
053⫶ super.toggle(forceEnabled);
054⫶
055⫶ this.outputElement.disabled = !forceEnabled || !this.outputElement.disabled;
056⫶ }
057⫶
058⫶ convertToBase64(data) {
059⫶ let binary = '';
060⫶
061⫶ for (let i = 0; i < data.length; i++) {
062⫶ binary += String.fromCharCode(data.charCodeAt(i) & 0xff);
063⫶ }
064⫶
065⫶ return btoa(binary);
066⫶ }
067⫶}


code_samples/ai_actions/config/packages/ibexa_admin_ui.yaml


code_samples/ai_actions/config/packages/ibexa_admin_ui.yaml

docs/ai_actions/extend_ai_actions.md@328:``` yaml
docs/ai_actions/extend_ai_actions.md@329:[[= include_file('code_samples/ai_actions/config/packages/ibexa_admin_ui.yaml') =]]
docs/ai_actions/extend_ai_actions.md@330:```

001⫶ibexa:
002⫶ system:
003⫶ admin_group:
004⫶ admin_ui_forms:
005⫶ content_edit:
006⫶ form_templates:
007⫶ - { template: '@ibexadesign/admin/ui/fieldtype/edit/form_fields_binary_ai.html.twig', priority: -10 } }


code_samples/ai_actions/config/services.yaml


code_samples/ai_actions/config/services.yaml

docs/ai_actions/extend_ai_actions.md@50:``` yaml
docs/ai_actions/extend_ai_actions.md@51:[[= include_file('code_samples/ai_actions/config/services.yaml', 25, 28) =]]
docs/ai_actions/extend_ai_actions.md@52:```

001⫶ App\Command\AddMissingAltTextCommand:
002⫶ arguments:
003⫶ $projectDir: '%kernel.project_dir%'

docs/ai_actions/extend_ai_actions.md@121:``` yaml
docs/ai_actions/extend_ai_actions.md@122:[[= include_file('code_samples/ai_actions/config/services.yaml', 28, 33) =]]
docs/ai_actions/extend_ai_actions.md@123:```

001⫶ App\AI\Handler\LLaVATextToTextActionHandler:
002⫶ tags:
003⫶ - { name: ibexa.ai.action.handler, priority: 0 }
004⫶ - { name: ibexa.ai.action.handler.text_to_text, priority: 0 }

docs/ai_actions/extend_ai_actions.md@141:``` yaml
docs/ai_actions/extend_ai_actions.md@142:[[= include_file('code_samples/ai_actions/config/services.yaml', 34, 41) =]]
docs/ai_actions/extend_ai_actions.md@143:```

001⫶ app.connector_ai.action_configuration.handler.llava_text_to_text.form_mapper.options:
002⫶ class: Ibexa\Bundle\ConnectorAi\Form\FormMapper\ActionConfiguration\ActionHandlerOptionsFormMapper
003⫶ arguments:
004⫶ $formType: 'App\Form\Type\TextToTextOptionsType'
005⫶ tags:
006⫶ - name: ibexa.connector_ai.action_configuration.form_mapper.options
007⫶ type: !php/const \App\AI\Handler\LLaVaTextToTextActionHandler::IDENTIFIER

docs/ai_actions/extend_ai_actions.md@154:``` yaml
docs/ai_actions/extend_ai_actions.md@155:[[= include_file('code_samples/ai_actions/config/services.yaml', 64, 66) =]]
docs/ai_actions/extend_ai_actions.md@156:```

001⫶ Ibexa\Contracts\ConnectorAi\ActionConfiguration\OptionsFormatterInterface:
002⫶ alias: Ibexa\ConnectorAi\ActionConfiguration\JsonOptionsFormatter

docs/ai_actions/extend_ai_actions.md@180:``` yaml
docs/ai_actions/extend_ai_actions.md@181:[[= include_file('code_samples/ai_actions/config/services.yaml', 42, 50) =]]
docs/ai_actions/extend_ai_actions.md@182:```

001⫶ App\AI\ActionType\TranscribeAudioActionType:
002⫶ arguments:
003⫶ $actionHandlers: !tagged_iterator
004⫶ tag: app.connector_ai.action.handler.audio_to_text
005⫶ default_index_method: getIdentifier
006⫶ index_by: key
007⫶ tags:
008⫶ - { name: ibexa.ai.action.type, identifier: !php/const \App\AI\ActionType\TranscribeAudioActionType::IDENTIFIER }

docs/ai_actions/extend_ai_actions.md@218:``` yaml
docs/ai_actions/extend_ai_actions.md@219:[[= include_file('code_samples/ai_actions/config/services.yaml', 51, 58) =]]
docs/ai_actions/extend_ai_actions.md@220:```

001⫶ app.connector_ai.action_configuration.handler.transcribe_audio.form_mapper.options:
002⫶ class: Ibexa\Bundle\ConnectorAi\Form\FormMapper\ActionConfiguration\ActionTypeOptionsFormMapper
003⫶ arguments:
004⫶ $formType: 'App\Form\Type\TranscribeAudioOptionsType'
005⫶ tags:
006⫶ - name: ibexa.connector_ai.action_configuration.form_mapper.action_type_options
007⫶ type: !php/const \App\AI\ActionType\TranscribeAudioActionType::IDENTIFIER

docs/ai_actions/extend_ai_actions.md@234:``` yaml
docs/ai_actions/extend_ai_actions.md@235:[[= include_file('code_samples/ai_actions/config/services.yaml', 59, 63) =]]
docs/ai_actions/extend_ai_actions.md@236:```

001⫶ App\AI\Handler\WhisperAudioToTextActionHandler:
002⫶ tags:
003⫶ - { name: ibexa.ai.action.handler, priority: 0 }
004⫶ - { name: app.connector_ai.action.handler.audio_to_text, priority: 0 }

docs/ai_actions/extend_ai_actions.md@252:``` yaml
docs/ai_actions/extend_ai_actions.md@253:[[= include_file('code_samples/ai_actions/config/services.yaml', 68, 72) =]]
docs/ai_actions/extend_ai_actions.md@254:```

001⫶ App\AI\REST\Input\Parser\TranscribeAudio:
002⫶ parent: Ibexa\Rest\Server\Common\Parser
003⫶ tags:
004⫶ - { name: ibexa.rest.input.parser, mediaType: application/vnd.ibexa.api.ai.TranscribeAudio }

docs/ai_actions/extend_ai_actions.md@279:``` yaml
docs/ai_actions/extend_ai_actions.md@280:[[= include_file('code_samples/ai_actions/config/services.yaml', 73, 76) =]]
docs/ai_actions/extend_ai_actions.md@281:```

001⫶ App\AI\REST\Output\Resolver\AudioTextResolver:
002⫶ tags:
003⫶ - { name: ibexa.ai.action.mime_type, key: application/vnd.ibexa.api.ai.AudioText }

docs/ai_actions/extend_ai_actions.md@290:``` yaml
docs/ai_actions/extend_ai_actions.md@291:[[= include_file('code_samples/ai_actions/config/services.yaml', 77, 81) =]]
docs/ai_actions/extend_ai_actions.md@292:```

001⫶ App\AI\REST\Output\ValueObjectVisitor\AudioText:
002⫶ parent: Ibexa\Contracts\Rest\Output\ValueObjectVisitor
003⫶ tags:
004⫶ - { name: ibexa.rest.output.value_object.visitor, type: App\AI\REST\Value\AudioText }


code_samples/ai_actions/src/AI/Action/TranscribeAudioAction.php


code_samples/ai_actions/src/AI/Action/TranscribeAudioAction.php

docs/ai_actions/extend_ai_actions.md@205:``` php
docs/ai_actions/extend_ai_actions.md@206:[[= include_file('code_samples/ai_actions/src/AI/Action/TranscribeAudioAction.php') =]]
docs/ai_actions/extend_ai_actions.md@207:```

001⫶<?php
002⫶
003⫶declare(strict_types=1);
004⫶
005⫶namespace App\AI\Action;
006⫶
007⫶use App\AI\DataType\Audio;
008⫶use Ibexa\Contracts\ConnectorAi\Action\Action;
009⫶
010⫶final class TranscribeAudioAction extends Action
011⫶{
012⫶ private Audio $audio;
013⫶
014⫶ public function __construct(Audio $audio)
015⫶ {
016⫶ $this->audio = $audio;
017⫶ }
018⫶
019⫶ public function getParameters(): array
020⫶ {
021⫶ return [];
022⫶ }
023⫶
024⫶ public function getInput(): Audio
025⫶ {
026⫶ return $this->audio;
027⫶ }
028⫶
029⫶ public function getActionTypeIdentifier(): string
030⫶ {
031⫶ return 'transcribe_audio';
032⫶ }
033⫶}


code_samples/ai_actions/src/AI/ActionType/TranscribeAudioActionType.php


code_samples/ai_actions/src/AI/ActionType/TranscribeAudioActionType.php

docs/ai_actions/extend_ai_actions.md@176:``` php
docs/ai_actions/extend_ai_actions.md@177:[[= include_file('code_samples/ai_actions/src/AI/ActionType/TranscribeAudioActionType.php') =]]
docs/ai_actions/extend_ai_actions.md@178:```

001⫶<?php
002⫶
003⫶declare(strict_types=1);
004⫶
005⫶namespace App\AI\ActionType;
006⫶
007⫶use App\AI\Action\TranscribeAudioAction;
008⫶use App\AI\DataType\Audio;
009⫶use Ibexa\Contracts\ConnectorAi\Action\DataType\Text;
010⫶use Ibexa\Contracts\ConnectorAi\ActionInterface;
011⫶use Ibexa\Contracts\ConnectorAi\ActionType\ActionTypeInterface;
012⫶use Ibexa\Contracts\ConnectorAi\DataType;
013⫶use Ibexa\Contracts\Core\Exception\InvalidArgumentException;
014⫶
015⫶final class TranscribeAudioActionType implements ActionTypeInterface
016⫶{
017⫶ public const IDENTIFIER = 'transcribe_audio';
018⫶
019⫶ /** @var iterable<\Ibexa\Contracts\ConnectorAi\Action\ActionHandlerInterface> */
020⫶ private iterable $actionHandlers;
021⫶
022⫶ /** @param iterable<\Ibexa\Contracts\ConnectorAi\Action\ActionHandlerInterface> $actionHandlers*/
023⫶ public function __construct(iterable $actionHandlers)
024⫶ {
025⫶ $this->actionHandlers = $actionHandlers;
026⫶ }
027⫶
028⫶ public function getIdentifier(): string
029⫶ {
030⫶ return self::IDENTIFIER;
031⫶ }
032⫶
033⫶ public function getName(): string
034⫶ {
035⫶ return 'Transcribe audio';
036⫶ }
037⫶
038⫶ public function getInputIdentifier(): string
039⫶ {
040⫶ return Audio::getIdentifier();
041⫶ }
042⫶
043⫶ public function getOutputIdentifier(): string
044⫶ {
045⫶ return Text::getIdentifier();
046⫶ }
047⫶
048⫶ public function getOptions(): array
049⫶ {
050⫶ return [];
051⫶ }
052⫶
053⫶ public function createAction(DataType $input, array $parameters = []): ActionInterface
054⫶ {
055⫶ if (!$input instanceof Audio) {
056⫶ throw new InvalidArgumentException(
057⫶ 'audio',
058⫶ 'expected \App\AI\DataType\Audio type, ' . get_debug_type($input) . ' given.'
059⫶ );
060⫶ }
061⫶
062⫶ return new TranscribeAudioAction($input);
063⫶ }
064⫶
065⫶ public function getActionHandlers(): iterable
066⫶ {
067⫶ return $this->actionHandlers;
068⫶ }
069⫶}


code_samples/ai_actions/src/AI/DataType/Audio.php


code_samples/ai_actions/src/AI/DataType/Audio.php

docs/ai_actions/extend_ai_actions.md@199:``` php
docs/ai_actions/extend_ai_actions.md@200:[[= include_file('code_samples/ai_actions/src/AI/DataType/Audio.php') =]]
docs/ai_actions/extend_ai_actions.md@201:```

001⫶<?php
002⫶
003⫶declare(strict_types=1);
004⫶
005⫶namespace App\AI\DataType;
006⫶
007⫶use Ibexa\Contracts\ConnectorAi\DataType;
008⫶
009⫶/**
010⫶ * @implements DataType<string>
011⫶ */
012⫶final class Audio implements DataType
013⫶{
014⫶ /** @var non-empty-array<string> */
015⫶ private array $base64;
016⫶
017⫶ /**
018⫶ * @param non-empty-array<string> $base64
019⫶ */
020⫶ public function __construct(array $base64)
021⫶ {
022⫶ $this->base64 = $base64;
023⫶ }
024⫶
025⫶ public function getBase64(): string
026⫶ {
027⫶ return reset($this->base64);
028⫶ }
029⫶
030⫶ public function getList(): array
031⫶ {
032⫶ return $this->base64;
033⫶ }
034⫶
035⫶ public static function getIdentifier(): string
036⫶ {
037⫶ return 'audio';
038⫶ }
039⫶}


code_samples/ai_actions/src/AI/Handler/LLaVaTextToTextActionHandler.php


code_samples/ai_actions/src/AI/Handler/LLaVaTextToTextActionHandler.php

docs/ai_actions/extend_ai_actions.md@117:``` php hl_lines="21 29-31 34-69 71-74"
docs/ai_actions/extend_ai_actions.md@118:[[= include_file('code_samples/ai_actions/src/AI/Handler/LLaVaTextToTextActionHandler.php') =]]
docs/ai_actions/extend_ai_actions.md@119:```

001⫶<?php
002⫶
003⫶declare(strict_types=1);
004⫶
005⫶namespace App\AI\Handler;
006⫶
007⫶use Ibexa\Contracts\ConnectorAi\Action\ActionHandlerInterface;
008⫶use Ibexa\Contracts\ConnectorAi\Action\DataType\Text;
009⫶use Ibexa\Contracts\ConnectorAi\Action\Response\TextResponse;
010⫶use Ibexa\Contracts\ConnectorAi\Action\TextToText\Action as TextToTextAction;
011⫶use Ibexa\Contracts\ConnectorAi\ActionInterface;
012⫶use Ibexa\Contracts\ConnectorAi\ActionResponseInterface;
013⫶use Symfony\Contracts\HttpClient\HttpClientInterface;
014⫶
015⫶final class LLaVaTextToTextActionHandler implements ActionHandlerInterface
016⫶{
017⫶ private HttpClientInterface $client;
018⫶
019⫶ private string $host;
020⫶
021⫸ public const IDENTIFIER = 'LLaVATextToText';
022⫶
023⫶ public function __construct(HttpClientInterface $client, string $host = 'http://localhost:8080')
024⫶ {
025⫶ $this->client = $client;
026⫶ $this->host = $host;
027⫶ }
028⫶
029⫸ public function supports(ActionInterface $action): bool
030⫸ {
031⫸ return $action instanceof TextToTextAction;
032⫶ }
033⫶
034⫸ public function handle(ActionInterface $action, array $context = []): ActionResponseInterface
035⫸ {
036⫸ /** @var \Ibexa\Contracts\ConnectorAi\Action\DataType\Text */
037⫸ $input = $action->getInput();
038⫸ $text = $this->sanitizeInput($input->getText());
039⫸
040⫸ $systemMessage = $action->hasActionContext() ? $action->getActionContext()->getActionHandlerOptions()->get('system_prompt', '') : '';
041⫸
042⫸ $response = $this->client->request(
043⫸ 'POST',
044⫸ sprintf('%s/v1/chat/completions', $this->host),
045⫸ [
046⫸ 'headers' => [
047⫸ 'Authorization: Bearer no-key',
048⫸ ],
049⫸ 'json' => [
050⫸ 'model' => 'LLaMA_CPP',
051⫸ 'messages' => [
052⫸ (object)[
053⫸ 'role' => 'system',
054⫸ 'content' => $systemMessage,
055⫸ ],
056⫸ (object)[
057⫸ 'role' => 'user',
058⫸ 'content' => $text,
059⫸ ],
060⫸ ],
061⫸ 'temperature' => 0.7,
062⫸ ],
063⫸ ]
064⫸ );
065⫸
066⫸ $output = strip_tags(json_decode($response->getContent(), true)['choices'][0]['message']['content']);
067⫸
068⫸ return new TextResponse(new Text([$output]));
069⫸ }
070⫶
071⫸ public static function getIdentifier(): string
072⫸ {
073⫸ return self::IDENTIFIER;
074⫸ }
075⫶
076⫶ private function sanitizeInput(string $text): string
077⫶ {
078⫶ return str_replace(["\n", "\r"], ' ', $text);
079⫶ }
080⫶}


code_samples/ai_actions/src/AI/Handler/WhisperAudioToTextActionHandler.php


code_samples/ai_actions/src/AI/Handler/WhisperAudioToTextActionHandler.php

docs/ai_actions/extend_ai_actions.md@230:``` php hl_lines="34-37 52-55"
docs/ai_actions/extend_ai_actions.md@231:[[= include_file('code_samples/ai_actions/src/AI/Handler/WhisperAudioToTextActionHandler.php') =]]
docs/ai_actions/extend_ai_actions.md@232:```

001⫶<?php
002⫶
003⫶declare(strict_types=1);
004⫶
005⫶namespace App\AI\Handler;
006⫶
007⫶use App\AI\ActionType\TranscribeAudioActionType;
008⫶use Ibexa\Contracts\ConnectorAi\Action\ActionHandlerInterface;
009⫶use Ibexa\Contracts\ConnectorAi\Action\DataType\Text;
010⫶use Ibexa\Contracts\ConnectorAi\Action\Response\TextResponse;
011⫶use Ibexa\Contracts\ConnectorAi\ActionInterface;
012⫶use Ibexa\Contracts\ConnectorAi\ActionResponseInterface;
013⫶use Symfony\Component\Process\Exception\ProcessFailedException;
014⫶use Symfony\Component\Process\Process;
015⫶
016⫶final class WhisperAudioToTextActionHandler implements ActionHandlerInterface
017⫶{
018⫶ private const TIMESTAMP_FORMAT = '/^\[\d{2}:\d{2}\.\d{3} --> \d{2}:\d{2}\.\d{3}]\s*/';
019⫶
020⫶ public function supports(ActionInterface $action): bool
021⫶ {
022⫶ return $action->getActionTypeIdentifier() === TranscribeAudioActionType::IDENTIFIER;
023⫶ }
024⫶
025⫶ public function handle(ActionInterface $action, array $context = []): ActionResponseInterface
026⫶ {
027⫶ /** @var \App\AI\DataType\Audio $input */
028⫶ $input = $action->getInput();
029⫶
030⫶ $path = $this->saveInputToFile($input->getBase64());
031⫶
032⫶ $arguments = ['whisper'];
033⫶
034⫸ $language = $action->getRuntimeContext()?->get('languageCode');
035⫸ if ($language !== null) {
036⫸ $arguments[] = sprintf('--language=%s', substr($language, 0, 2));
037⫸ }
038⫶
039⫶ $arguments[] = '--output_format=txt';
040⫶ $arguments[] = $path;
041⫶
042⫶ $process = new Process($arguments);
043⫶ $process->run();
044⫶
045⫶ if (!$process->isSuccessful()) {
046⫶ unlink($path);
047⫶ throw new ProcessFailedException($process);
048⫶ }
049⫶
050⫶ $output = $process->getOutput();
051⫶
052⫸ $includeTimestamps = $action->getActionContext()
053⫸ ?->getActionTypeOptions()
054⫸ ->get('include_timestamps', false)
055⫸ ?? false;
056⫶
057⫶ if (!$includeTimestamps) {
058⫶ $output = $this->removeTimestamps($output);
059⫶ }
060⫶
061⫶ unlink($path);
062⫶
063⫶ return new TextResponse(new Text([$output]));
064⫶ }
065⫶
066⫶ public static function getIdentifier(): string
067⫶ {
068⫶ return 'whisper_audio_to_text';
069⫶ }
070⫶
071⫶ private function removeTimestamps(string $text): string
072⫶ {
073⫶ $lines = explode(PHP_EOL, $text);
074⫶
075⫶ $processedLines = array_map(static function (string $line): string {
076⫶ return preg_replace(self::TIMESTAMP_FORMAT, '', $line) ?? '';
077⫶ }, $lines);
078⫶
079⫶ return implode(PHP_EOL, $processedLines);
080⫶ }
081⫶
082⫶ private function saveInputToFile(string $audioEncodedInBase64): string
083⫶ {
084⫶ $filename = uniqid('audio');
085⫶ $path = sys_get_temp_dir() . \DIRECTORY_SEPARATOR . $filename;
086⫶ file_put_contents($path, base64_decode($audioEncodedInBase64));
087⫶
088⫶ return $path;
089⫶ }
090⫶}


code_samples/ai_actions/src/AI/REST/Input/Parser/TranscribeAudio.php


code_samples/ai_actions/src/AI/REST/Input/Parser/TranscribeAudio.php

docs/ai_actions/extend_ai_actions.md@248:``` php
docs/ai_actions/extend_ai_actions.md@249:[[= include_file('code_samples/ai_actions/src/AI/REST/Input/Parser/TranscribeAudio.php') =]]
docs/ai_actions/extend_ai_actions.md@250:```

001⫶<?php
002⫶
003⫶declare(strict_types=1);
004⫶
005⫶namespace App\AI\REST\Input\Parser;
006⫶
007⫶use App\AI\DataType\Audio as AudioDataType;
008⫶use App\AI\REST\Value\TranscribeAudioAction;
009⫶use Ibexa\ConnectorAi\REST\Input\Parser\Action;
010⫶use Ibexa\Contracts\ConnectorAi\Action\RuntimeContext;
011⫶use Ibexa\Contracts\Rest\Input\ParsingDispatcher;
012⫶use Ibexa\Rest\Input\BaseParser;
013⫶
014⫶final class TranscribeAudio extends BaseParser
015⫶{
016⫶ public const AUDIO_KEY = 'Audio';
017⫶ public const BASE64_KEY = 'base64';
018⫶
019⫶ /** @param array<mixed> $data */
020⫶ public function parse(array $data, ParsingDispatcher $parsingDispatcher): TranscribeAudioAction
021⫶ {
022⫶ $this->assertInputIsValid($data);
023⫶ $runtimeContext = $this->getRuntimeContext($data);
024⫶
025⫶ return new TranscribeAudioAction(
026⫶ new AudioDataType([$data[self::AUDIO_KEY][self::BASE64_KEY]]),
027⫶ $runtimeContext
028⫶ );
029⫶ }
030⫶
031⫶ /** @param array<mixed> $data */
032⫶ private function assertInputIsValid(array $data): void
033⫶ {
034⫶ if (!array_key_exists(self::AUDIO_KEY, $data)) {
035⫶ throw new \InvalidArgumentException('Missing audio key');
036⫶ }
037⫶
038⫶ if (!array_key_exists(self::BASE64_KEY, $data[self::AUDIO_KEY])) {
039⫶ throw new \InvalidArgumentException('Missing base64 key');
040⫶ }
041⫶ }
042⫶
043⫶ /**
044⫶ * @param array<string, mixed> $data
045⫶ */
046⫶ private function getRuntimeContext(array $data): RuntimeContext
047⫶ {
048⫶ return new RuntimeContext(
049⫶ $data[Action::RUNTIME_CONTEXT_KEY] ?? []
050⫶ );
051⫶ }
052⫶}


code_samples/ai_actions/src/AI/REST/Output/Resolver/AudioTextResolver.php


code_samples/ai_actions/src/AI/REST/Output/Resolver/AudioTextResolver.php

docs/ai_actions/extend_ai_actions.md@275:``` php
docs/ai_actions/extend_ai_actions.md@276:[[= include_file('code_samples/ai_actions/src/AI/REST/Output/Resolver/AudioTextResolver.php') =]]
docs/ai_actions/extend_ai_actions.md@277:```

001⫶<?php
002⫶
003⫶declare(strict_types=1);
004⫶
005⫶namespace App\AI\REST\Output\Resolver;
006⫶
007⫶use App\AI\REST\Value\AudioText;
008⫶use Ibexa\ConnectorAi\REST\Output\ResolverInterface;
009⫶use Ibexa\Contracts\ConnectorAi\ActionResponseInterface;
010⫶
011⫶final class AudioTextResolver implements ResolverInterface
012⫶{
013⫶ public function getRestValue(
014⫶ ActionResponseInterface $actionResponse
015⫶ ): AudioText {
016⫶ return new AudioText(
017⫶ $actionResponse->getOutput()
018⫶ );
019⫶ }
020⫶}


code_samples/ai_actions/src/AI/REST/Output/ValueObjectVisitor/AudioText.php


code_samples/ai_actions/src/AI/REST/Output/ValueObjectVisitor/AudioText.php

docs/ai_actions/extend_ai_actions.md@286:``` php
docs/ai_actions/extend_ai_actions.md@287:[[= include_file('code_samples/ai_actions/src/AI/REST/Output/ValueObjectVisitor/AudioText.php') =]]
docs/ai_actions/extend_ai_actions.md@288:```

001⫶<?php
002⫶
003⫶declare(strict_types=1);
004⫶
005⫶namespace App\AI\REST\Output\ValueObjectVisitor;
006⫶
007⫶use Ibexa\Contracts\Rest\Output\Generator;
008⫶use Ibexa\Contracts\Rest\Output\ValueObjectVisitor;
009⫶use Ibexa\Contracts\Rest\Output\Visitor;
010⫶
011⫶final class AudioText extends ValueObjectVisitor
012⫶{
013⫶ private const OBJECT_IDENTIFIER = 'AudioText';
014⫶
015⫶ public function visit(Visitor $visitor, Generator $generator, $data): void
016⫶ {
017⫶ $mediaType = 'ai.' . self::OBJECT_IDENTIFIER;
018⫶ $text = $data->getOutput();
019⫶
020⫶ $generator->startObjectElement(self::OBJECT_IDENTIFIER, $mediaType);
021⫶ $visitor->setHeader('Content-Type', $generator->getMediaType($mediaType));
022⫶
023⫶ $visitor->visitValueObject($text);
024⫶
025⫶ $generator->endObjectElement(self::OBJECT_IDENTIFIER);
026⫶ }
027⫶}


code_samples/ai_actions/src/AI/REST/Value/AudioText.php


code_samples/ai_actions/src/AI/REST/Value/AudioText.php

docs/ai_actions/extend_ai_actions.md@268:``` php
docs/ai_actions/extend_ai_actions.md@269:[[= include_file('code_samples/ai_actions/src/AI/REST/Value/AudioText.php') =]]
docs/ai_actions/extend_ai_actions.md@270:```

001⫶<?php
002⫶
003⫶declare(strict_types=1);
004⫶
005⫶namespace App\AI\REST\Value;
006⫶
007⫶use Ibexa\ConnectorAi\REST\Value\RestActionResponse;
008⫶
009⫶final class AudioText extends RestActionResponse
010⫶{
011⫶}


code_samples/ai_actions/src/AI/REST/Value/TranscribeAudioAction.php


code_samples/ai_actions/src/AI/REST/Value/TranscribeAudioAction.php

docs/ai_actions/extend_ai_actions.md@258:``` php
docs/ai_actions/extend_ai_actions.md@259:[[= include_file('code_samples/ai_actions/src/AI/REST/Value/TranscribeAudioAction.php') =]]
docs/ai_actions/extend_ai_actions.md@260:```

001⫶<?php
002⫶
003⫶declare(strict_types=1);
004⫶
005⫶namespace App\AI\REST\Value;
006⫶
007⫶use App\AI\DataType\Audio;
008⫶use Ibexa\Contracts\ConnectorAi\Action\RuntimeContext;
009⫶
010⫶final class TranscribeAudioAction
011⫶{
012⫶ private Audio $input;
013⫶
014⫶ private RuntimeContext $runtimeContext;
015⫶
016⫶ public function __construct(
017⫶ Audio $input,
018⫶ RuntimeContext $runtimeContext
019⫶ ) {
020⫶ $this->input = $input;
021⫶ $this->runtimeContext = $runtimeContext;
022⫶ }
023⫶
024⫶ public function getInput(): Audio
025⫶ {
026⫶ return $this->input;
027⫶ }
028⫶
029⫶ public function getRuntimeContext(): RuntimeContext
030⫶ {
031⫶ return $this->runtimeContext;
032⫶ }
033⫶}


code_samples/ai_actions/src/Command/ActionConfigurationCreateCommand.php


code_samples/ai_actions/src/Command/ActionConfigurationCreateCommand.php

docs/ai_actions/extend_ai_actions.md@79:``` php hl_lines="3 17"
docs/ai_actions/extend_ai_actions.md@80:[[= include_file('code_samples/ai_actions/src/Command/ActionConfigurationCreateCommand.php', 59, 76) =]]
docs/ai_actions/extend_ai_actions.md@81:```

001⫶ $refineTextActionType = $this->actionTypeRegistry->getActionType('refine_text');
002⫶
003⫸ $actionConfigurationCreateStruct = new ActionConfigurationCreateStruct('rewrite_casual');
004⫶
005⫶ $actionConfigurationCreateStruct->setType($refineTextActionType);
006⫶ $actionConfigurationCreateStruct->setName('eng-GB', 'Rewrite in casual tone');
007⫶ $actionConfigurationCreateStruct->setDescription('eng-GB', 'Rewrites the text using a casual tone');
008⫶ $actionConfigurationCreateStruct->setActionHandler('openai-text-to-text');
009⫶ $actionConfigurationCreateStruct->setActionHandlerOptions(new ArrayMap([
010⫶ 'max_tokens' => 4000,
011⫶ 'temperature' => 1,
012⫶ 'prompt' => 'Rewrite this content to improve readability. Preserve meaning and crucial information but use casual language accessible to a broader audience.',
013⫶ 'model' => 'gpt-4-turbo',
014⫶ ]));
015⫶ $actionConfigurationCreateStruct->setEnabled(true);
016⫶
017⫸ $this->actionConfigurationService->createActionConfiguration($actionConfigurationCreateStruct);

docs/ai_actions/extend_ai_actions.md@90:``` php hl_lines="7-8"
docs/ai_actions/extend_ai_actions.md@91:[[= include_file('code_samples/ai_actions/src/Command/ActionConfigurationCreateCommand.php', 77, 85) =]]
docs/ai_actions/extend_ai_actions.md@92:```

001⫶ $action = new RefineTextAction(new Text([
002⫶ <<<TEXT
003⫶ Proteins differ from one another primarily in their sequence of amino acids, which is dictated by the nucleotide sequence of their genes,
004⫶ and which usually results in protein folding into a specific 3D structure that determines its activity.
005⫶TEXT
006⫶ ]));
007⫸ $actionConfiguration = $this->actionConfigurationService->getActionConfiguration('rewrite_casual');
008⫸ $actionResponse = $this->actionService->execute($action, $actionConfiguration)->getOutput();


code_samples/ai_actions/src/Command/AddMissingAltTextCommand.php


code_samples/ai_actions/src/Command/AddMissingAltTextCommand.php

docs/ai_actions/extend_ai_actions.md@16:``` php
docs/ai_actions/extend_ai_actions.md@17:[[= include_file('code_samples/ai_actions/src/Command/AddMissingAltTextCommand.php', 101, 120) =]]
docs/ai_actions/extend_ai_actions.md@18:```

001⫶ $action = new GenerateAltTextAction(new Image([$imageEncodedInBase64]));
002⫶
003⫶ $action->setRuntimeContext(new RuntimeContext(['languageCode' => $languageCode]));
004⫶ $action->setActionContext(
005⫶ new ActionContext(
006⫶ new ActionConfigurationOptions(['default_locale_fallback' => 'en']), // System context
007⫶ new ActionConfigurationOptions(['max_lenght' => 100]), // Action Type options
008⫶ new ActionConfigurationOptions( // Action Handler options
009⫶ [
010⫶ 'prompt' => 'Generate the alt text for this image in less than 100 characters.',
011⫶ 'temperature' => 0.7,
012⫶ 'max_tokens' => 4096,
013⫶ 'model' => 'gpt-4o-mini',
014⫶ ]
015⫶ )
016⫶ )
017⫶ );
018⫶
019⫶ $output = $this->actionService->execute($action)->getOutput();

docs/ai_actions/extend_ai_actions.md@46:``` php hl_lines="87 100-125"
docs/ai_actions/extend_ai_actions.md@47:[[= include_file('code_samples/ai_actions/src/Command/AddMissingAltTextCommand.php') =]]
docs/ai_actions/extend_ai_actions.md@48:```

001⫶<?php
002⫶
003⫶declare(strict_types=1);
004⫶
005⫶namespace App\Command;
006⫶
007⫶use Ibexa\Contracts\ConnectorAi\Action\ActionContext;
008⫶use Ibexa\Contracts\ConnectorAi\Action\DataType\Image;
009⫶use Ibexa\Contracts\ConnectorAi\Action\DataType\Text;
010⫶use Ibexa\Contracts\ConnectorAi\Action\GenerateAltTextAction;
011⫶use Ibexa\Contracts\ConnectorAi\Action\RuntimeContext;
012⫶use Ibexa\Contracts\ConnectorAi\ActionConfiguration\ActionConfigurationOptions;
013⫶use Ibexa\Contracts\ConnectorAi\ActionServiceInterface;
014⫶use Ibexa\Contracts\Core\Repository\ContentService;
015⫶use Ibexa\Contracts\Core\Repository\FieldTypeService;
016⫶use Ibexa\Contracts\Core\Repository\PermissionResolver;
017⫶use Ibexa\Contracts\Core\Repository\UserService;
018⫶use Ibexa\Contracts\Core\Repository\Values\Content\ContentList;
019⫶use Ibexa\Contracts\Core\Repository\Values\Content\Query\Criterion\ContentTypeIdentifier;
020⫶use Ibexa\Contracts\Core\Repository\Values\Content\Query\Criterion\DateMetadata;
021⫶use Ibexa\Contracts\Core\Repository\Values\Content\Query\Criterion\Operator;
022⫶use Ibexa\Contracts\Core\Repository\Values\Filter\Filter;
023⫶use Ibexa\Core\FieldType\Image\Value;
024⫶use Symfony\Component\Console\Command\Command;
025⫶use Symfony\Component\Console\Input\InputArgument;
026⫶use Symfony\Component\Console\Input\InputInterface;
027⫶use Symfony\Component\Console\Output\OutputInterface;
028⫶
029⫶final class AddMissingAltTextCommand extends Command
030⫶{
031⫶ protected static $defaultName = 'app:add-alt-text';
032⫶
033⫶ private const IMAGE_FIELD_IDENTIFIER = 'image';
034⫶
035⫶ private ContentService $contentService;
036⫶
037⫶ private PermissionResolver $permissionResolver;
038⫶
039⫶ private UserService $userService;
040⫶
041⫶ private FieldTypeService $fieldTypeService;
042⫶
043⫶ private ActionServiceInterface $actionService;
044⫶
045⫶ private string $projectDir;
046⫶
047⫶ public function __construct(
048⫶ ContentService $contentService,
049⫶ PermissionResolver $permissionResolver,
050⫶ UserService $userService,
051⫶ FieldTypeService $fieldTypeService,
052⫶ ActionServiceInterface $actionService,
053⫶ string $projectDir
054⫶ ) {
055⫶ parent::__construct();
056⫶ $this->contentService = $contentService;
057⫶ $this->permissionResolver = $permissionResolver;
058⫶ $this->userService = $userService;
059⫶ $this->fieldTypeService = $fieldTypeService;
060⫶ $this->actionService = $actionService;
061⫶ $this->projectDir = $projectDir;
062⫶ }
063⫶
064⫶ protected function configure(): void
065⫶ {
066⫶ $this->addArgument('user', InputArgument::OPTIONAL, 'Login of the user executing the actions', 'admin');
067⫶ }
068⫶
069⫶ protected function execute(InputInterface $input, OutputInterface $output): int
070⫶ {
071⫶ $this->setUser($input->getArgument('user'));
072⫶
073⫶ $modifiedImages = $this->getModifiedImages();
074⫶ $output->writeln(sprintf('Found %d modified image in the last 24h', $modifiedImages->getTotalCount()));
075⫶
076⫶ /** @var \Ibexa\Contracts\Core\Repository\Values\Content\Content $content */
077⫶ foreach ($modifiedImages as $content) {
078⫶ /** @var ?Value $value */
079⫶ $value = $content->getFieldValue(self::IMAGE_FIELD_IDENTIFIER);
080⫶
081⫶ if ($value === null || !$this->shouldGenerateAltText($value)) {
082⫶ $output->writeln(sprintf('Image %s has the image field empty or the alternative text is already specified. Skipping.', $content->getName()));
083⫶ continue;
084⫶ }
085⫶
086⫶ $contentUpdateStruct = $this->contentService->newContentUpdateStruct();
087⫸ $value->alternativeText = $this->getSuggestedAltText($this->convertImageToBase64($value->uri), $content->getDefaultLanguageCode());
088⫶ $contentUpdateStruct->setField(self::IMAGE_FIELD_IDENTIFIER, $value);
089⫶
090⫶ $updatedContent = $this->contentService->updateContent(
091⫶ $this->contentService->createContentDraft($content->getContentInfo())->getVersionInfo(),
092⫶ $contentUpdateStruct
093⫶ );
094⫶ $this->contentService->publishVersion($updatedContent->getVersionInfo());
095⫶ }
096⫶
097⫶ return Command::SUCCESS;
098⫶ }
099⫶
100⫸ private function getSuggestedAltText(string $imageEncodedInBase64, string $languageCode): string
101⫸ {
102⫸ $action = new GenerateAltTextAction(new Image([$imageEncodedInBase64]));
103⫸
104⫸ $action->setRuntimeContext(new RuntimeContext(['languageCode' => $languageCode]));
105⫸ $action->setActionContext(
106⫸ new ActionContext(
107⫸ new ActionConfigurationOptions(['default_locale_fallback' => 'en']), // System context
108⫸ new ActionConfigurationOptions(['max_lenght' => 100]), // Action Type options
109⫸ new ActionConfigurationOptions( // Action Handler options
110⫸ [
111⫸ 'prompt' => 'Generate the alt text for this image in less than 100 characters.',
112⫸ 'temperature' => 0.7,
113⫸ 'max_tokens' => 4096,
114⫸ 'model' => 'gpt-4o-mini',
115⫸ ]
116⫸ )
117⫸ )
118⫸ );
119⫸
120⫸ $output = $this->actionService->execute($action)->getOutput();
121⫸
122⫸ assert($output instanceof Text);
123⫸
124⫸ return $output->getText();
125⫸ }
126⫶
127⫶ private function convertImageToBase64(?string $uri): string
128⫶ {
129⫶ $file = file_get_contents($this->projectDir . \DIRECTORY_SEPARATOR . 'public' . \DIRECTORY_SEPARATOR . $uri);
130⫶ if ($file === false) {
131⫶ throw new \RuntimeException('Cannot read file');
132⫶ }
133⫶
134⫶ return 'data:image/jpeg;base64,' . base64_encode($file);
135⫶ }
136⫶
137⫶ private function getModifiedImages(): ContentList
138⫶ {
139⫶ $filter = (new Filter())
140⫶ ->withCriterion(
141⫶ new DateMetadata(DateMetadata::MODIFIED, Operator::GTE, strtotime('-1 day'))
142⫶ )
143⫶ ->andWithCriterion(new ContentTypeIdentifier('image'));
144⫶
145⫶ return $this->contentService->find($filter);
146⫶ }
147⫶
148⫶ private function shouldGenerateAltText(Value $value): bool
149⫶ {
150⫶ return $this->fieldTypeService->getFieldType('ezimage')->isEmptyValue($value) === false &&
151⫶ $value->isAlternativeTextEmpty();
152⫶ }
153⫶
154⫶ private function setUser(string $userLogin): void
155⫶ {
156⫶ $this->permissionResolver->setCurrentUserReference($this->userService->loadUserByLogin($userLogin));
157⫶ }
158⫶}


code_samples/ai_actions/src/Form/Type/TextToTextOptionsType.php


code_samples/ai_actions/src/Form/Type/TextToTextOptionsType.php

docs/ai_actions/extend_ai_actions.md@137:``` php hl_lines="16-20"
docs/ai_actions/extend_ai_actions.md@138:[[= include_file('code_samples/ai_actions/src/Form/Type/TextToTextOptionsType.php') =]]
docs/ai_actions/extend_ai_actions.md@139:```

001⫶<?php
002⫶
003⫶declare(strict_types=1);
004⫶
005⫶namespace App\Form\Type;
006⫶
007⫶use Symfony\Component\Form\AbstractType;
008⫶use Symfony\Component\Form\Extension\Core\Type\TextareaType;
009⫶use Symfony\Component\Form\FormBuilderInterface;
010⫶use Symfony\Component\OptionsResolver\OptionsResolver;
011⫶
012⫶final class TextToTextOptionsType extends AbstractType
013⫶{
014⫶ public function buildForm(FormBuilderInterface $builder, array $options): void
015⫶ {
016⫸ $builder->add('system_prompt', TextareaType::class, [
017⫸ 'required' => true,
018⫸ 'disabled' => $options['translation_mode'],
019⫸ 'label' => 'System message',
020⫸ ]);
021⫶ }
022⫶
023⫶ public function configureOptions(OptionsResolver $resolver): void
024⫶ {
025⫶ $resolver->setDefaults([
026⫶ 'translation_domain' => 'app_ai',
027⫶ 'translation_mode' => false,
028⫶ ]);
029⫶
030⫶ $resolver->setAllowedTypes('translation_mode', 'bool');
031⫶ }
032⫶}


code_samples/ai_actions/src/Form/Type/TranscribeAudioOptionsType.php


code_samples/ai_actions/src/Form/Type/TranscribeAudioOptionsType.php

docs/ai_actions/extend_ai_actions.md@214:``` php hl_lines="16-20"
docs/ai_actions/extend_ai_actions.md@215:[[= include_file('code_samples/ai_actions/src/Form/Type/TranscribeAudioOptionsType.php') =]]
docs/ai_actions/extend_ai_actions.md@216:```

001⫶<?php
002⫶
003⫶declare(strict_types=1);
004⫶
005⫶namespace App\Form\Type;
006⫶
007⫶use Symfony\Component\Form\AbstractType;
008⫶use Symfony\Component\Form\Extension\Core\Type\CheckboxType;
009⫶use Symfony\Component\Form\FormBuilderInterface;
010⫶use Symfony\Component\OptionsResolver\OptionsResolver;
011⫶
012⫶final class TranscribeAudioOptionsType extends AbstractType
013⫶{
014⫶ public function buildForm(FormBuilderInterface $builder, array $options): void
015⫶ {
016⫸ $builder->add('include_timestamps', CheckboxType::class, [
017⫸ 'required' => false,
018⫸ 'disabled' => $options['translation_mode'],
019⫸ 'label' => 'Include timestamps',
020⫸ ]);
021⫶ }
022⫶
023⫶ public function configureOptions(OptionsResolver $resolver): void
024⫶ {
025⫶ $resolver->setDefaults([
026⫶ 'translation_domain' => 'app_ai',
027⫶ 'translation_mode' => false,
028⫶ ]);
029⫶
030⫶ $resolver->setAllowedTypes('translation_mode', 'bool');
031⫶ }
032⫶}


code_samples/ai_actions/src/Query/Search.php


code_samples/ai_actions/src/Query/Search.php

docs/search/ai_actions_search_reference/action_configuration_criteria.md@19:``` php
docs/search/ai_actions_search_reference/action_configuration_criteria.md@20:[[= include_file('code_samples/ai_actions/src/Query/Search.php') =]]
docs/search/ai_actions_search_reference/action_configuration_criteria.md@21:```

001⫶<?php
002⫶
003⫶declare(strict_types=1);
004⫶
005⫶use Ibexa\Contracts\ConnectorAi\ActionConfiguration\ActionConfigurationQuery;
006⫶use Ibexa\Contracts\ConnectorAi\ActionConfiguration\Query\Criterion;
007⫶use Ibexa\Contracts\ConnectorAi\ActionConfiguration\Query\SortClause;
008⫶use Ibexa\Contracts\CoreSearch\Values\Query\AbstractSortClause;
009⫶use Ibexa\Contracts\CoreSearch\Values\Query\Criterion\FieldValueCriterion;
010⫶
011⫶$query = new ActionConfigurationQuery(
012⫶ new Criterion\LogicalAnd(
013⫶ new Criterion\Enabled(),
014⫶ new Criterion\LogicalOr(
015⫶ new Criterion\Name('Casual', FieldValueCriterion::COMPARISON_STARTS_WITH),
016⫶ new Criterion\Identifier('casual')
017⫶ )
018⫶ ),
019⫶ [
020⫶ new SortClause\Enabled(AbstractSortClause::SORT_DESC),
021⫶ new SortClause\Identifier(AbstractSortClause::SORT_ASC),
022⫶ ]
023⫶);
024⫶/** @var \Ibexa\Contracts\ConnectorAi\ActionConfigurationServiceInterface $actionConfigurationService */
025⫶$results = $actionConfigurationService->findActionConfigurations($query);

docs/search/ai_actions_search_reference/action_configuration_sort_clauses.md@14:``` php
docs/search/ai_actions_search_reference/action_configuration_sort_clauses.md@15:[[= include_file('code_samples/ai_actions/src/Query/Search.php') =]]
docs/search/ai_actions_search_reference/action_configuration_sort_clauses.md@16:```

001⫶<?php
002⫶
003⫶declare(strict_types=1);
004⫶
005⫶use Ibexa\Contracts\ConnectorAi\ActionConfiguration\ActionConfigurationQuery;
006⫶use Ibexa\Contracts\ConnectorAi\ActionConfiguration\Query\Criterion;
007⫶use Ibexa\Contracts\ConnectorAi\ActionConfiguration\Query\SortClause;
008⫶use Ibexa\Contracts\CoreSearch\Values\Query\AbstractSortClause;
009⫶use Ibexa\Contracts\CoreSearch\Values\Query\Criterion\FieldValueCriterion;
010⫶
011⫶$query = new ActionConfigurationQuery(
012⫶ new Criterion\LogicalAnd(
013⫶ new Criterion\Enabled(),
014⫶ new Criterion\LogicalOr(
015⫶ new Criterion\Name('Casual', FieldValueCriterion::COMPARISON_STARTS_WITH),
016⫶ new Criterion\Identifier('casual')
017⫶ )
018⫶ ),
019⫶ [
020⫶ new SortClause\Enabled(AbstractSortClause::SORT_DESC),
021⫶ new SortClause\Identifier(AbstractSortClause::SORT_ASC),
022⫶ ]
023⫶);
024⫶/** @var \Ibexa\Contracts\ConnectorAi\ActionConfigurationServiceInterface $actionConfigurationService */
025⫶$results = $actionConfigurationService->findActionConfigurations($query);


code_samples/ai_actions/templates/themes/admin/admin/ui/fieldtype/edit/form_fields_binary_ai.html.twig


code_samples/ai_actions/templates/themes/admin/admin/ui/fieldtype/edit/form_fields_binary_ai.html.twig

docs/ai_actions/extend_ai_actions.md@323:``` twig
docs/ai_actions/extend_ai_actions.md@324:[[= include_file('code_samples/ai_actions/templates/themes/admin/admin/ui/fieldtype/edit/form_fields_binary_ai.html.twig') =]]
docs/ai_actions/extend_ai_actions.md@325:```

001⫶{% extends '@ibexadesign/ui/field_type/edit/ezbinaryfile.html.twig' %}
002⫶
003⫶{% block ezbinaryfile_preview %}
004⫶ {{ parent() }}
005⫶
006⫶ {% set transcriptFieldIdentifier = 'transcript' %}
007⫶ {% set fieldTypeIdentifiers = form.parent.parent.vars.value|keys %}
008⫶
009⫶ {% if transcriptFieldIdentifier in fieldTypeIdentifiers %}
010⫶ {% set module_id = 'TranscribeAudio' %}
011⫶ {% set ai_config_id = 'transcribe_audio' %}
012⫶ {% set container_selector = '.ibexa-edit-content' %}
013⫶ {% set input_selector = '.ibexa-field-edit-preview__action--preview' %}
014⫶ {% s...*[Comment body truncated]*

@mnocon
Copy link
Contributor Author

mnocon commented Dec 19, 2024

@julitafalcondusza @adriendupuis thank you for the suggestions! I've applied them in 8010fce and in 82cc5c4

Copy link
Contributor

@adriendupuis adriendupuis left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Few more remarks but nothing mandatory.


# Extend AI Actions

By extending AI Actions, you can make regular content management and editing tasks more appealing and less demanding.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe a link back for readers directly arriving here from a search result.

Suggested change
By extending AI Actions, you can make regular content management and editing tasks more appealing and less demanding.
By extending [AI Actions](ai_actions_guide.md), you can make regular content management and editing tasks more appealing and less demanding.

Comment on lines +79 to +82
<<<TEXT
Proteins differ from one another primarily in their sequence of amino acids, which is dictated by the nucleotide sequence of their genes,
and which usually results in protein folding into a specific 3D structure that determines its activity.
TEXT
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would inject less white spaces.

Suggested change
<<<TEXT
Proteins differ from one another primarily in their sequence of amino acids, which is dictated by the nucleotide sequence of their genes,
and which usually results in protein folding into a specific 3D structure that determines its activity.
TEXT
<<<TEXT
Proteins differ from one another primarily in their sequence of amino acids, which is dictated by the nucleotide sequence of their genes,
and which usually results in protein folding into a specific 3D structure that determines its activity.
TEXT

You can pass one directly to the `ActionServiceInterface::execute()` method:

``` php hl_lines="7-8"
[[= include_file('code_samples/ai_actions/src/Command/ActionConfigurationCreateCommand.php', 77, 85) =]]
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would have prefer a rewrite of the previous example.

Suggested change
[[= include_file('code_samples/ai_actions/src/Command/ActionConfigurationCreateCommand.php', 77, 85) =]]
$action = new GenerateAltTextAction(new Image([$imageEncodedInBase64]));
$actionConfiguration = $this->actionConfigurationService->getActionConfiguration('TODO_alt_text');
$output = $this->actionService->execute($action, $actionConfiguration)->getOutput();

But no big deal, reader should connect the dot and see how this could have been used in the previous example.

Use the `Ibexa\Bundle\ConnectorAi\Form\FormMapper\ActionConfiguration\ActionHandlerOptionsFormMapper` class together with the `ibexa.connector_ai.action_configuration.form_mapper.options` service tag to make it part of the Action Handler options form.
Pass the Action Handler identifier (`LLaVATextToText`) as the type when tagging the service.

The Action Handler and Action Type options are rendered in the back office using the built-in Twig option formatter.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe a screenshot of the default rendering of this example form.

mnocon and others added 3 commits December 19, 2024 16:01
* Generated PHP API Reference

* Apply suggestions from code review

Co-authored-by: julitafalcondusza <117284672+julitafalcondusza@users.noreply.github.com>
Co-authored-by: Adrien Dupuis <61695653+adriendupuis@users.noreply.github.com>

* Manual review changes

---------

Co-authored-by: julitafalcondusza <117284672+julitafalcondusza@users.noreply.github.com>
Co-authored-by: Adrien Dupuis <61695653+adriendupuis@users.noreply.github.com>
@mnocon mnocon merged commit 3db6364 into master Dec 19, 2024
1 of 3 checks passed
@mnocon mnocon deleted the IBX-8689-extending branch December 19, 2024 21:20
mnocon added a commit that referenced this pull request Dec 19, 2024
* Added extending AI documentation

* Apply suggestions from code review

Co-authored-by: julitafalcondusza <117284672+julitafalcondusza@users.noreply.github.com>
Co-authored-by: Adrien Dupuis <61695653+adriendupuis@users.noreply.github.com>

* Manual review changes

* Generated PHP reference (#2539)

* Generated PHP API Reference

* Apply suggestions from code review

Co-authored-by: julitafalcondusza <117284672+julitafalcondusza@users.noreply.github.com>
Co-authored-by: Adrien Dupuis <61695653+adriendupuis@users.noreply.github.com>

* Manual review changes

---------

Co-authored-by: julitafalcondusza <117284672+julitafalcondusza@users.noreply.github.com>
Co-authored-by: Adrien Dupuis <61695653+adriendupuis@users.noreply.github.com>

* Regenerated reference

---------

Co-authored-by: julitafalcondusza <117284672+julitafalcondusza@users.noreply.github.com>
Co-authored-by: Adrien Dupuis <61695653+adriendupuis@users.noreply.github.com>
@mnocon mnocon mentioned this pull request Jan 2, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants